.. _mlmodule:

Machine Learning
================

.. toctree::
   algorithms
   machinelearningmodel
   engines
   inferenceyamlconfigchangelog
   :hidden:
   :maxdepth: 2

The machine learning (ML) plugin contains all generic algorithms related to the execution of machine learning models (in particular neural networks) in the ImFusion software.
It also adds extra functionalities related to pre- and post-processing of models input and output, data augmentation, image sampling, etc.

This plugin contains the main pieces of the machine learning framework but is not self-sufficient. In order to execute ML models, additional plugins that natively integrate external frameworks are required.
This can be done either in C++ or in Python (provided you have the ``imfusion-sdk`` python package). In either case, the resulting plugin will be available to the ImFusionSuite/SDK.
See the page :ref:`Machine Learning Model<machinelearningmodel>` for more details.

The following ML frameworks are currently supported for C++ development:

+----------------+--------------------------+----------------+-----------------+
| Framework      | Windows                  | Linux          | MacOS Arm       |
+================+==========================+================+=================+
| Torch          | Yes: CPU, CUDA           | Yes: CPU, CUDA | Yes: CPU, MPS   |
+----------------+--------------------------+----------------+-----------------+
| ONNX Runtime   | Yes: CPU, CUDA, DirectML | Yes: CPU, CUDA | No              |
+----------------+--------------------------+----------------+-----------------+
| TensorRT       | Yes                      | Yes            | No              |
+----------------+--------------------------+----------------+-----------------+

.. note:: Previous versions of the ImFusion Suite might include CaffePlugin, TensorFlowPlugin or TOMAATPlugin; they are no longer supported.


In Python, we currently offer implementations for the following frameworks as part of the ``imfusion-sdk`` package:

* `PyTorch <https://pypi.org/project/torch/>`_
* `ONNX Runtime <https://pypi.org/project/onnxruntime/>`_
* `Core ML <https://pypi.org/project/coremltools/>`_
* `OpenVINO <https://pypi.org/project/openvino/>`_


.. note:: The ``imfusion-sdk`` does not ship the ``pytorch``, ``onnxruntime``, ``coreml`` and so on. In order to use the corresponding framework, you need to install it (i.e. with ``pip``). In this sense, we do not specify which device is supported since this depends on the python package flavor which is installed (i.e. ``torch-gpu`` or ``onnxruntime-silicon`` and so on).

The main algorithm of the ML plugin is :ref:`Run MachineLearning Model<ML.RunModel>`.
The :ref:`Operations Sequence<ML.OperationsSequence>` algorithm is also useful to check the pre- and post-processing of a model.
All other algorithms are considered experimental or internal.